Method and Apparatus for Recognition of Inhaler Actuation

ABSTRACT

A medication confirmation method and apparatus for confirming administration of medication employing an inhalable medication administration apparatus. The method of an embodiment of the invention includes the steps of capturing one or more video sequences of a user administering medication employing the inhalable medication administration apparatus, storing the captured one or more video sequences, capturing one or more audio sequences of the user administering medication employing the inhalable medication administration apparatus and storing the captured one or more audio sequences. At least one of the stored video sequences and at least one of the stored audio sequences are then analyzed to confirm that the user has properly administered the medication.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is Continuation in Part Application of U.S. patentapplication Ser. No. 13/235,387, filed Sep. 15, 2011 to Hanina et al.,titled APPARATUS AND METHOD FOR RECOGNITION OF PATIENT ACTIVITIES,currently pending, which is in turn a non-provisional application thatclaims the benefit of U.S. Provisional Patent Application Ser. No.61/498,576, filed Jun. 19, 2011 to Hanina et al., titled APPARATUS ANDMETHOD FOR RECOGNITION OF PATIENT ACTIVITIES, the entire contentsthereof being incorporated herein by reference. This application is alsoa Continuation in Part Application of U.S. patent application Ser. No.12/815,037, filed Jun. 14, 2010 to Hanina et al., titled APPARATUS ANDMETHOD FOR RECOGNITION OF PATIENT ACTIVITIES WHEN OBTAINING PROTOCOLADHERENCE DATA, currently pending, which claims the benefit of U.S.Provisional Patent Application Ser. No. 61/331,872 filed May 6, 2010, toHanina et al., titled APPARATUS AND METHOD FOR RECOGNITION OF PATIENTACTIVITIES WHEN OBTATINING PROTOCOL ADHERENCE DATA, the entire contentsof each of these applications also being incorporated herein byreference.

FIELD OF THE INVENTION

This invention relates generally to patient compliance in medicationadministration protocol scenarios, and more particularly to an apparatusand method for the collection, analysis and transmission of data relatedto patient movements related to such medication administration in orderto improve adherence with prescribed drug protocols in accordancetherewith. The invention further relates to a tracking and guidancetraining system for individuals taking prescribed medication, preferablyemploying computer vision and activity tracking, and may also provideincentives such as reminding a patient of the benefits of a particularmedication, providing the patient monetary points or other remunerationon the screen, and one or more reminders to take an appropriatemedication. The invention finally preferably relates to the use ofcomputer vision and activity recognition for determination of properadministration of injectable, inhaler-based, or other non-pillmedication administration in which proper position and action sequenceis important, confirming proper identity of the medication device, andproper actual ingestion, inhalation or injection of the medication, butis equally applicable to pill based medication administration sequences.

BACKGROUND OF THE INVENTION

Dr Lars Osterberg, M.D. and Dr, Terence Blaschke have reported in theNew England Journal of Medicine, Adherence to Medication, (N Engl J Med2005; 353:487-97) 2005 an alarming lack of adherence to requiredmedication protocol, further noting that while the average rates ofadherence in clinical trials is categorized as “high”, this number stillcomprises only rates of 43 to 78 percent. Most importantly, the authorsnote “The ability of physicians to recognize nonadherence is poor, andinterventions to improve adherence have had mixed results.” Adherence,p. 487. The authors conclude “Poor adherence to medication regimens iscommon, contributing to substantial worsening of disease, death andincreased healthcare costs.” Adherence, p. 494. The Trend Repot Series,2008 Patient Adherence Update: New Approaches for Success, October 2008,report similar discouraging statistics. This broad range may possiblycontribute to the public confidence in the FDA approval process and theimportance of continued surveillance of a drug throughout the process.Furthermore, it may help to explain why, according to the Journal of theAmerican Medical Association (JAMA May 1, 2002), one out of every fivenew drugs that comes to market in the US is found to have serious orlife-threatening adverse effects—unknown or undisclosed at the time ofapproval. It is against this backdrop of poor adherence, and potentialdanger to patients, that the present invention operates.

It has been widely recognized that methods and systems for insuringproper medication ingestion or administration by individuals are veryimportant in defending against unnecessary sickness, deaths and otherproblems. Giving instructions and then letting patients fend forthemselves has been shown not to work particularly well. This is becauseit is not only the improper ingestion of medicines that is the primarycause of medical danger. Rather, an overall lack of sufficient patientguidance is also part of the problem. Further, the inability to confirma proper prescription regimen being provided to a user in the firstplace may cause a number of other problems with the use of suchmedication. As has been shown in regards to various public healthmedication administration situation, such as administration oftuberculosis medication by the WHO, Directly Observed Treatment (DOT)improves compliance of patients. Global Tuberculosis Control: A ShortUpdate to the 2009 Report, World Health Organization, 2009. As is shownin this report, funding for implementing DOT programs is high. Thus, theability to implement such a program with less of a financial burdenwould be desirable.

Traditionally, participants attend introductions and follow ups forclinical trials in-person. Other patients outside of the clinical trialsetting attempting to adhere to a particular medication protocolsimilarly are given a prescription and a particular set of instructionsfrom a prescribing medical provider or prescribing doctor, and thencompliance is measured at a next visit with that prescribingprofessional through traditional methods of counting unused medication,and patient interviews. Thus, data collection is similarly limited topatient visits, rather than on a daily basis. These old methods such aspatient questioning and medication counting have been proven to beinadequate measures of adherence and offer no information on dose timingand drug holidays (omission of medication for three or more sequentialdays).

Compliance technologies can increase the statistical power of clinicaltrials. Through the use of such technology, clinical events can beprecisely linked to medication use history. Captured data can be linkedto other sources such as EDC, patient diaries and data collected by thephysician. Technologies can create many possibilities for remote visitsand data capture. While smart packaging technologies exist such asRFID-enabled computer chip technology, smart blister packs and MEMS caps(microprocessor in a bottle cap), they are: a) invasive and need to bephysically attached to the medications; b) are non-conclusive regardingcompliance—a patient may activate the technology without ingestion ofthe medication; c) remain largely unadopted in clinical trials by thepharmaceutical and biotech companies due to their high cost; and d) takea longer time to implement. Further, electronic patient diaries allowfor ease of entry of data by a patient. These diaries, however, arestill subject to issues related to compliance with medication adherence.Thus, even if a patient is meticulous about entering information intothe diary, and thus complying with the requirements for data entry,there is still no guarantee that they are properly taking medication atprescribed times. Additionally, none of these techniques allow for themonitoring of non-pill related medications, such as the user of inhalersor injectable medication. These medication delivery systems requireadditional monitoring, as know when administration took place is notsufficient. Rather, positioning and other technique issues maysubstantially affect the proper delivery and efficacy of particularmedications.

Particularly related to inhalable medication, determining properinhalation of medication has been difficult. A number of differentsystems have been proposed and employed to make such a determination.There are several electronic monitors that have been reported in theliterature for use with Metered Dose Inhalers (MDIs) or dry powderinhalers (DPIs). The “nebulizer chronolog” device and the “DoserClinical Trials” device have been used with MDIs. The nebulizerchronolog is a microprocessor device built into the sleeve housing anMDI; it records the date and time of each inhaler actuation, byactivation of a microswitch. The Doser Clinical Trials device isdescribed as an pressure-activated device, also used with MDIs. It is around, flat device secured to the top of the canister, and it recordsonly the number of daily uses a period of 45 days. A similar MDIelectromechanicalcounter has also been employed The ElectronicDiskhalerallows monitoring of the Diskhaler DPI, by drug blister piercing andairflow through the device. A similar device, the Turbohaler InhalationComputer has been used with the Turbohaler DPI, known as Turbuhaler inthe United States. An electronic monitor has also been reported for theDiskus DPI.

It should be noted that not all electronic monitors actual inhalation ofmedication by patients. With the chronolog, medication can be sprayedinto the air, or the switch flicked manually. The Electronic recordsboth blister perforation and airflow, which some indication thatinhalation occurred following loading. Researchers have investigatedadherence with aerosol therapy, using the nebulizer chronolog, incomparison with canister weighing and patient self-report with a groupof patients with chronic obstructive pulmonary disease (COPD). Thisstudy found that both canister weights and self-report overestimatedadherence with prescribed therapy among patients who were not informedof the nebulizer chronolog's recording ability

Even these electronic systems, as recognized by J M Foster et al. (JAsthma. 2012 August; 49(6):657-62. Epub 2012 Jun. 2), are not withoutcriticism, however. As indicated, they actually measure compliance withactuation of the metered-dose inhaler and not direct information aboutmedication ingestion.

Jo Carol et al. stated that “The most reliable method for researchpurposes, although not practical in a clinical setting, may be acombination approach that includes pill counts, patient self-report, andelectronic monitoring.” (Carol J. et al, Patterns to AntiretroviralMedication, The Value of Electronic Monitoring, AIDS, 17 (12), pp 1,763-767, October 2003. To date, technologies alone have only been usedto monitor compliance rather than to encourage it. Furthermore, therehas been no comprehensive system provided that allows for the managementof multiple patients and multiple patient populations. While currenttechnology may allow poor compliers to be recognized, as will bedescribed below, the proposed apparatus and method of the presentinvention will help to encourage pharmaceutical compliance with non-pilldelivered medications, and tackle some of the problems that areencountered in the clinical trial process in particular, and themedication protocol monitoring problem in general.

A number of systems exist that provide instructions to a user regardingwhen to take a medication and records when the user indicates that amedication has been taken. U.S. Pat. No. 7,359,214 describes such asystem. A device is provided that provides instruction to a patientregarding medications to take. Furthermore, the system may provide amethod for determining that the prescription is appropriate given thepatient's conditions, and other medications he or she may already betaking The system may monitor the dispensing of medicine in accordancewith a predetermined treatment protocol. While such a system providesmany improvements for easing a burden on the patient, this systemsuffers in many ways and in particular in ways relevant to theadministration of clinical trials and other active patient monitoring ofmedication adherence.

Most importantly, this system provides no mechanism for actuallyconfirming that a patient is in fact properly administering requiredmedication as required in a clinical drug trial, as prescribed by aprescribing physician in the case where adherence to a particularregimen may prove to be critical to efficacy of the prescriptionregimen, in various public health scenarios, in situations where failureto keep up a prescription regimen can potentially harm a population as awhole, such as the generation of antibiotic-resistant bacteria strains,in various disease management scenarios, or in home care situationswhere maintaining proper control of administering healthcareprofessionals is critical. Further, while the system may be sufficientfor one who is in full possession of their mental faculties, anyindividual who may have difficulty following directions, or one who isactively avoiding medication may still not be taking required medicationafter it is dispensed. Thus, participants may be forgetful, visuallyimpaired, or otherwise do not believe in the benefit of taking suchmedication, and may thus not properly log medication administration.Furthermore, as it applies only to pill based oral medication, thesystem requires preloading of various medications into a dispenser, andthus likely requires regular visits by an administering manager to besure appropriate medications are in fact properly loaded therein. It issurely possible that an inexperienced user may place incorrectmedications into the device, or may somehow provide incorrect dosagesinto the device. Additionally, for potentially more complex regimens,there is no method provided for insuring that a user is able to followsuch a protocol, and to thereafter confirm that the user has in facttaken all required medications in accordance with any providedinstructions or the like, or has taken the medications according to oneor more specifications or followed suggested procedures. Furthermore,this system is expensive and requires constant maintenance to confirmthat the various mechanical parts are in working order. Finally, asnoted above, the system has no application to non-pill basedmedications.

U.S. patent application Ser. No. 11/839,723, filed Aug. 16, 2007, titledMobile Wireless Medication Management System provides a medicationmanagement system employing mobile devices and an imaging technology sothat a user is able to show a pill to be taken to the system, and thesystem can then identify the medication. Patient histories are availableto an administrator, including various vital signs as measured by thesystem. Images may also be taken of the patient, provider, medicationcontainer or the like. While the system professes to ensure adherence toa protocol, the system only provides such help if requested by a user.There is in fact no particular manner in which to ensure actualadherence or ingestion of the medication, or the relationship ofadherence to the efficacy or safety of the drug over time. Whenrequiring adherence to a predetermined protocol for a clinical trial,this is particularly relevant. Similarly, there is no mention ofnon-pill based medications.

While adherence to medication in general is poor, requirements for useof inhalable medications, such as metered dose inhalers (MDI) and dryinhalers, have an increased burden in that simply confirming actuationof such an inhaler is insufficient. It is important to use inhalerscorrectly to get the full dosage and benefit from the medicine. By usingthe MDI correctly, medication has a better chance to reach the smallairways, increasing medication effectiveness. Using a good technique canalso help reduce the side effects of medications. However, use of propertechnique with such inhalers is difficult to instruct and monitor inthat between 28% and 68% of patients do not use metered-dose inhalers orpowder inhalers well enough to benefit from the prescribed medication,and 39%-67% of nurses, doctors, and respiratory therapists are unable toadequately describe or perform critical steps for using inhalers. Evenif patients are able to demonstrate correct technique duringconsultation with a health professional, they may not maintain thisstandard at other times. Improvement in patient compliance with therapywill require better doctor-patient communication, improved patienteducation, the tailoring of therapy to the individual and possible novelstrategies such as offering feedback to the patients on further theirlevel of compliance.

Additionally, existing systems fail to maintain an audit trail for postadministration review by a medical official or other clinical trialadministrator, and further cannot therefore confirm confirmation ofproper medication administration or population management.

Therefore, it would be desirable to provide an apparatus that overcomesthe drawbacks of the prior art.

SUMMARY OF THE INVENTION

In U.S. patent application Ser. No. 12/620,686, filed Nov. 18, 2009,titled Method and Apparatus for Verification of MedicationAdministration Adherence; currently pending, U.S. patent applicationSer. No. 12/646,383, filed Dec. 23, 2009, titled Method and Apparatusfor Verification of Clinical Trial Adherence, currently pending; U.S.patent application Ser. No. 12/646,603, filed Dec. 23, 2009, titledMethod and Apparatus for Management of Clinical Trials, currentlypending; and U.S. patent application Ser. No. 12/728,721, filed Mar. 22,2010, titled Apparatus and Method for Collection of Protocol AdherenceData, currently pending, the contents of these four applications beingincorporated herein by reference, the inventors of the present inventionhave proposed a system, method and apparatus that allow for completecontrol and verification of adherence to a prescribed medicationprotocol or machine or apparatus use in a clinical trial setting,whether in a health care provider's care, or when self administered in ahomecare situation by a patient.

These applications present the only medication management system thatmay determine whether a user is actually following a protocol, provideadditional assistance to a user, starting with instructions, videoinstructions, and the like, and moving up to contact from a medicationadministrator if it is determined that the user would need suchassistance in any medical adherence situation, including clinical trialsettings, home care settings, healthcare administration locations, suchas nursing homes, clinics, hospitals and the like, and in clinical trialsettings.

The inventive solution, in accordance with one or more embodiments ofthe present invention, may provide a webcam software solution fordistribution by medical professionals to provide a training system fortraining patients to properly administer their inhalable, injectable, orother non-pill based medication, to automate direct observation ofmedication administration of inhalable, injectable or other medications,and to provide an audit trail of medication adherence and patientbehavior. The inventive system may visually and audibly track medicationadherence to inhalable, injectable and other medication during trainingand actual medication administration in clinical trials or othermedication administration scenarios on webcam-enabled laptops, tablets,smartphones and other platforms without real time human supervision. Theinventive system may visually and audibly recognize a fixed series ofactions, each comprising part of the medication administration process.

In accordance with an embodiment of the present invention, a motioncapture procedure for capturing motion information related to theadministration of injectable, inhaler-based, or other non-pill basedmedication, may be utilized in accordance with one or more of theinventions noted in the above-referenced applications. Suchadministration information preferably includes confirming a correctdevice, medication inserted therein, and then confirming actualingestion, inhalation or injection of the medication by the patient.Therefore, in accordance with an embodiment of the present invention, amethod and apparatus may be provided for analyzing captured patientmotion data, transmitting such captured patient motion data to a remotelocation (or processing such captured information locally, in whole orin part), receiving information from a remote location (or from aprocessor maintained locally) and providing information to the patientas preferred in accordance with the present invention.

Further in accordance with an embodiment of the present invention, oneor more predetermined motion sequences may be determined and correlatedto one or more corresponding medication administration instructions foradministration of injectable, inhaler-based, or other non-pill basedmedication. These predetermined motion sequences may be provided as anumber of generic motion sequences, as one or more customized motionsequences, or a combination of both, and preferably may include trackingof a medication administration device, thus confirming in real timeingestion, inhalation or injection of medication by the user. The groupof predetermined motion sequences may comprise a motion language thatmay be applied to one or more different medication administrationsequences, including injectable medication administration, inhaler-basedmedication administration, other non-pill based medicationadministration and the like, and other healthcare related processes,such as hand washing or the like, medication administration personnelacting as prescribed or the like, thus allowing for an easy to programgeneric medication administration sequence, but also allow forcustomization where appropriate and necessary. These programmed motionsequences may then be applied in accordance with the inventions noted inthe applications above.

In yet another embodiment of the invention, one or more methods may beprovided for confirming that a user is properly performing one or moreof these predetermined motion sequences. Thus, as a user positionsthemselves or an object before an image capture device, a display mayindicate to the user whether the position, distance, and/or skew andangle are correct. If not, the user is preferably provided withindications as to how to correct any one or more of these issues. Themotion sequences may include capture of use of an inhaler, injectabledevice or the like. Accumulation of such motion sequences may beemployed to count total dosages, remind the user when to refill theirmedication prescription, warn when the user might be running low onmedication, and order refills in advance based upon the number of timesthe user has used the device.

In a still further embodiment, one or more audio cues may also beemployed. Thus, for example, in the case of an inhaler-based medication,audio monitoring of sound from both the inhaler and patient may beperformed and used to further confirm that the patient has in factproperly administered the medication. Therefore, not only maypositioning of the inhaler in the correct location and relative angle beconfirmed, but activation of one or more inhaler mechanisms, and theinhaling of the medication by the patient upon such activation may alsobe confirmed. Micro movements may also be determined, so that smallmovements associated with medication administration in general, andingestion, inhalation or injection in particular can be used to confirmactuation of a device, for example. Thus, in an embodiment of theinvention, fingers actuating a device, other small movements, audiosignals, and the like may be combined to provide greater confidence indevice actuation, and inhalation, ingestion or injection confirmation.Similar monitoring may also be performed with injectable and othernon-pill based medications.

The system in accordance with one or more of the various embodiments ofthe present invention may also be applicable to training situationswhere the user is provided with various feedback instructions related totraining to properly administer medication in a clinical trial or otherdisease management scenario. In accordance with various embodiments ofthe present invention, for example, when applied to an inhaler, apatient may be requested to shake the inhaler before use. When appliedto an injectable medication, the patient may be requested to confirmrefrigeration or confirm proper sanitization of an injectable tip withan alcohol swab or the like, confirm that the needle is not bent, orthat an injectable solution or medication has not changed color, orotherwise become spoiled in a manner that is visually detectable. Whileonline training and instructions may be available currently, theinteractive nature providing feedback to the user regarding their useand following of protocol is critical in improving adherence and patientaction.

Still other objects and advantages of the invention will in part beobvious and will in part be apparent from the specification anddrawings.

The invention accordingly comprises the several steps and the relationof one or more of such steps with respect to each of the others, and theapparatus embodying features of construction, combinations of elementsand arrangement of parts that are adapted to affect such steps, all asexemplified in the following detailed disclosure, and the scope of theinvention will be indicated in the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the invention, reference is made tothe following description and accompanying drawings, in which:

FIG. 1 is a flow chart diagram depicting a method in accordance with anembodiment of the invention;

FIG. 2 is a flowchart diagram depicting a video sequence generationmethod in accordance with an embodiment of the invention;

FIG. 3 is a flowchart diagram depicting a video sequence capture methodin accordance with an embodiment of the invention;

FIG. 4 is a depiction of a positioning process in accordance with anembodiment;

FIG. 5 is a depiction of another positioning process in accordance withan embodiment of the invention;

FIG. 6 is a depiction of yet another positioning process in accordancewith an embodiment of the invention;

FIG. 7 is a depiction of a motion tracking process in accordance with anembodiment of the invention;

FIG. 8 is a flowchart diagram depicting a combined video and audiodetection system for detecting proper medication administration of aninhaler in accordance with an embodiment of the invention;

FIG. 9 is a flowchart diagram depicting an audio signature recognitionsystem in accordance with an embodiment of the invention;

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The invention will now be described making reference to the followingdrawings in which like reference numbers denote like structure or steps.Referring to FIG. 1, a data flow overview in accordance with theoperation of an embodiment of the present invention is shown. Inaccordance with this embodiment of the invention, information about aparticular drug to be the subject of a clinical trial, to be employed ina public health or disease management situation, or the like, or othermedication administration program or prescription may be provided in adatabase 105, and existing industry medication information databases 110are preferably employed to access prescription, interaction,application, and other available information about any number ofproposed prescription and non-prescription medications and theirpossible interaction with the clinical trial or other medications.Further, patient medical records 115 may be used, and as will bedescribed below, can be used in conjunction with the industry medicalinformation and a medical professional's prescribing expertise toconfirm that a patient is a good candidate for such a clinical trial, ormedication administration program. These databases may be accessed in amanner known to one of ordinary skill in the art.

Once confirmed, a medication administration regimen in accordance withthe clinical trial or other prescription requirements such as in apublic health, medical practice environment or the like may beprescribed and entered into the system of the invention at 120. Onceentered into the system, a particular prescription regimen may cause aset of user instructions, various training sequences and the like 125 tobe generated and transmitted to an apparatus provided to a patient inaccordance with an embodiment of the invention for access to the systemof the invention. Such an apparatus may comprise a custom designed videoand audio capture, analysis and transmission apparatus, a smart phone orother mobile device including a camera or other video and audio captureapparatuses, a netbook, laptop computer, desktop computer, tablet deviceor the like, or other computing appliance allowing for the display ofinstructions to a patient, and allowing for the eventual capture,analysis and transmission of video, audio and other analysisinformation. When installing software on a user's own hardware system,it is preferred that the software detect and otherwise test or determinethat the hardware attempting to be utilized by the patient is sufficientto implement the invention and is sufficient to run a software packageprovided in accordance with the invention. Thus, the software may checkthat a camera includes sufficient resolution, that a memory of thedevice is of sufficient size to allow for sufficient captured videostorage, that audio may be properly captured, and that the transmissionsystem includes sufficient bandwidth to transmit and receive capturedvideo, audio, video instructions and the like.

In a clinical trial or other administration settings, patientinstructions and various training sequences may be varied for differentusers to determine the best set of instructions, or may be varied basedupon demographics, experience, or other factors that may requiredifferent types of instructions to be provided. It is furthercontemplated in accordance with an embodiment of the invention thatmultiple clinical trials or patient populations may be managed by amanager in accordance with the invention so that the inventioncontemplates a medication administration system that allows for a singlepoint of management for all clinical trials or patient management groupsassociated with a particular manager or the like. Such managementtechniques in accordance with the embodiment of the invention mayfurther be applied to various public health situations, diseasemanagement scenarios and the like.

Such user instructions and training sequences may include generalinstructions about the particular medication and/or device such as aninhaler or injectable device subject to the current trial or medicationadministration protocol, methods for administration, warnings about sideeffects, and concerns about drug interactions with common substances ormedications, or other medications prescribed to the patient by thesystem or by another medical service provider. It is contemplated inaccordance with an embodiment of the invention that such set of userinstructions may be interactive, allowing a user to view additionalinformation about such instructions or prescriptions as desired. Theseinstructions may comprise written, audio or video instructions providedto the user on a display of the user apparatus. These instruction mayfurther comprise interactive instructions provided to a user on adevice, adapting to the actions of a user, thus providing asophisticated interactive training apparatus and system. In an inhalablesystem, such visual guides may include advising the user when to inhaleand exhale, how long to hold their breath. The system may also advise ifa location is too noisy, or has poor lighting. It is furthercontemplated that such instructions may indicate one or more movementsequences to be associated with a corresponding one or more medicationadministration sequences. A more in-depth description of the informationprovided at step 125 is depicted in FIG. 2.

As is shown in FIG. 2, the generation and provision of user instructionsas set forth in step 125 first comprises the step of receiving asequence of required instruction steps at step 205. This sequence may bedetermined as described above in step 120. The system then may confirmwhether one or more of the instructions steps require the conveyance ofinformation to a patient at step 210. These conveyance steps maycomprise a more conventional instruction step, such as the display ofwritten information, comprise a more advanced instruction step, such asthe conveyance of audible information, video instructions or the like,or may comprise an interactive instruction step, such as an interactiveinstruction sequence displaying a desired sequence of information to apatient, and then monitoring and confirming whether the patent hasproperly administered the medication. Various feedback mechanisms may beprovided to allow the patient to try multiple times to perform properadministration, and may also provide varying encouragement orinstructions to confirm that administration training has been performedproperly. Thus, such an instruction and training sequence may includethe eventual capture of video, audio and other information from theuser. Therefore, at step 215, it may be determined whether one or moreof the instruction steps will require the capture of information fromthe user, thus comprising an advanced interactive training sequence.Thereafter, each of the training steps requiring capture of videoinformation from a user is confirmed at step 220. If no further videocapture is required, and therefore various training or other interactivesequences have been completed, processing for step 125 then ends at step250. If it is determined that the capture of video and/or audioinformation will be required at step 220 for the current training step,then processing passes to step 230, and various instructional video,audio and other sequences may be provided to the user in aninstructional sequence format.

Such interactive instructions sequence may be particularly applicable tothe use of an inhaler solution in that proper positioning and use of aninhaler is particularly important. As noted above, proper angling of theinhaler device, synchronization of inhalation and actuation of thedevice, and proper timing of maintaining the device and inhaled breathare all factors that contribute to the efficacy of an inhalablemedication. Training of patients to properly perform these actions, andfurther monitoring during each use, may improve the medicationexperience of the patient, allowing for improved outcomes using themedication.

After being shown a particular instructional sequence, preferablyapplicable to a particular step of a medication administration protocolsequence, then processing passes to step 235 where the user may beprompted to perform a particular action or sequence of movements. Theuser may request to be re-shown these sequences as many times asnecessary, and may also include audio or other instructions, so that theuser is provided with a training sequence, thereby reducing variabilityof future performance of that action. When preparing to perform theseactions, an alert system may be employed to warn the patient of anyissues that may interfere with the proper capture of video and/or audioinformation, as may take place similarly when actually administering themedication. Thus, the user may be encouraged to properly perform thesesequences, thus acting as an interactive training module.

Thus, the user may be notified if they are sitting in a manner in whichtheir actions cannot be properly captured, if they are blocked from thecamera, the light conditions are insufficient, if an object they areholding is in an improper location, or the like. As is shown in FIG. 4,a box 410 may be provided on a display viewable by a patient using thesystem. A representation of the patient's face may be shown in aposition relative to an optimal filming position for the use of, forexample, an inhaler for medication administration. Thus, while facialrepresentation 400a is properly positioned, facial representation 400bis positioned to the left of the box, while facial representation 400 cis positioned down and to the right of the box. A similar positioningsystem may be provided for an injectable medication, the position of apatient body part being provided in place of the facial positioningdescribed above. Thus, not only may proper positioning be determined,but use of the proper body part may also be confirmed. In practice, thebox may be made a red or other warning color until proper alignment isachieved (including if a user or desired user body part is notpositioned fully within a screen, the user is too close or far from thecamera, or for any other reason), at which time the box may change togreen or other appropriate color. Further, audio clues may also be givento the patient, such as increasing frequency beeping as the optimalposition is approached. Any other positioning indicators or the like mayalso be employed.

Thus, in accordance with an embodiment of the present invention to beemployed for inhaler medications, the user is preferably provided withimmediate feedback on their position and the ability of their actions tobe properly recorded and analyzed. As the user preferably interacts withthe system of this embodiment of the invention, such a scheme may beemployed to provide continuous feedback to the user, and thus indicatingwhether the system is able to properly capture and/or analyze theactions of the user. If time is passed and the user is unable toproperly position themselves, or to properly perform desired actions,additional guidance may be provided to the user in order to remedy sucha situation, including but not limited to directional indications, voicecommands, video images of proper technique, etc.

In addition to properly positioning the patient, proper positioning ofone or more objects, either absolutely or relative to another body part,may be determined, such as positioning an inhaler relative to the mouthand face of the user, an injectable medication delivery device relativeto the body part of the user to receive the injection, or the like forimaging and processing in accordance with an embodiment of theinvention. As is shown in FIG. 5, an inhaler 500 may be indicated asproperly positioned by a box 522, the box being green, for example, asin the description of FIG. 4. Such an object, however, is more likely tobe improperly positioned not only left to right and up to down, but alsoin distance to the imaging apparatus, in accordance with one or morelimitations of the imaging device, such as the resolution thereof, lowlight positions, and the like, and any affect such resolution might haveon the ability of the imaging device to identify shape, color text orother coding, or the like associated with the object being imaged. Thus,if positioned too far away from the imaging apparatus, a sequence ofboxes 510, 511, 512 and a small representation of inhaler 500 may beprovided to alert the user to move the inhaler closer. If the inhaler isnot only too far away, but off center, boxes 520, 521, 522 may beprovided to guide the user to move the inhaler into proper positionabsolutely and relative to the mouth and face of the user. Similarfunctionality may be provided for positioning an injectable apparatus orinhaler apparatus relative to a user body part to receive the injection,including relative angle and distance to the body part, or relative tothe mouth of the user when employing an inhaler apparatus. By properlypositioning such a device, the system may be employed to confirm theidentity of such a medication, employing shape, color, labeling, and thelike. Further, a real time tracking system may be employed that includesvisual guides to properly training and guide the user as to thedirection and angle of the device, such as an inhaler device, thusallowing for optimal usage. Such a real-time interactive system allowsfor advanced user interaction via a sophisticated user interface tocommunicate with, and adapt to, a user. In the case of an inhalerdevice, an interactive training system may therefore be provided beprovided to further insure, in the case of a metered dose inhaler, thatthe canister has been properly inserted in to the inhaler device, mayaid the user in checking the expiration date, if any, or may do soautomatically, wither through text recognition or through the use of abarcode reading system. The system may further be employed to requestand confirm the user actuates multiple inhalation doses, if necessary.

In addition to determining identity of the medication, such processingmay be used to determine safety of the apparatus, such as whether aninhaler or injectable device may have been damaged or tampered with.Further, the medication may be observed to determine any change in coloror other characteristic of the medication that may suggest spoilage,improper medication, counterfeit medication or the like. The apparatus,in accordance with an embodiment of the invention, may thus ask the userto move the inhaler or injectable device closer to or further away fromthe imaging apparatus, may change an ambient light sensitivity of theapparatus, or may otherwise change details of the image capture. Asnoted above, both color and audio prompting may be provided.

To the extent that positioning and orientation of the inhaler,injectable medication administrator or the like when being used isimportant, a similar system may be employed. As is shown in FIG. 6, aset of concentric circles 610 a-e (or other positioning indicator) maybe provided to aid in the positioning of an inhaler 600. A center circle610e may be provided with a solid center (not shown) upon properplacement of the inhaler. These circles may move as the boxes in FIG. 5,and may further use color and/or audio prompts to instruct the user.Further, as images of inhaler positions and orientations, or inhaler andhand positions and orientations, are to be captured and analyzed, thesystem may also preferably indicate not only proper positioning, butactual acquisition of a correct position and orientation sequence. Inaccordance with an additional embodiment of the invention, suchrecognizable positioning and orientation may further comprise a sequenceof gestures and apparatus movement and orientation employed to ensurethat the patient properly administers their medication. In accordancewith an administration process, as noted above, the patient may first betrained to show a particular medication administration device orapparatus in their hand to the camera for imaging and recognition. Thepatient may then be asked to place the apparatus at an appropriateadministration location, such as against the mouth in the case of aninhaler apparatus, or at a particular body part location in the case ofan injectable medication. Thereafter, actuation of the apparatus,through the process of monitoring movement and audible cues may beemployed. Thus, through a predetermined sequence of actions that arecaptured, imaged and analyzed, evidence of proper administration can berecorded and analyzed. Visual and audio prompts to aid the user inproperly positioning the inhalable device relative to the mouth of theuser may also be provided.

Furthermore, in accordance with one or more embodiments of theinvention, various additional aspects of medication and/oradministration may be checked and confirmed. Thus, the system may employsuch computer vision and activity recognition to determine a liquidcolor, liquid consistency or clarity, potential existence of particles,perhaps suggesting a spoiled medication, bubbles in the liquid,suggesting improper handling, in an injectable administration system.Through the use of the system, a number of administrations can betracked, and a liquid or other level may be used to confirm the count,thus potentially allowing for the addition ordering of furthermedication, or other counting of inhaler administrations without theneed for expensive inhaler units. Also, dosage settings, if applicableon an injectable pen or other apparatus may also be confirmed beforeadministration.

Furthermore, as is shown in FIG. 7, when tracking the movement of amedication administration apparatus 700, it is preferable to depict to apatient whether they are holding the apparatus at a correct orientation,when the apparatus is in transit, or positioned at the administrationsight. Thus, as is show in FIG. 7, an administration apparatus 700 isindicated to be reoriented from a horizontal to a vertical orientationthrough movement in the direction noted by arrows A. A set of guidancetracks 710 a, 710 b may be displayed to a patient and successiveapparatus positions and orientations may be superimposed thereon. As theapparatus moves along the proscribed path, concentric circles such asthose depicted in FIG. 6 may be employed to confirm proper location andorientation. Thus, in accordance with an embodiment of the invention, avirtual path may be shown to the user to ensure that the proper methodof medication administration is followed. As noted above, color and/oraudio sequences may also be employed. Similar positioning informationmay be processed relative to an injectable medication.

Therefore, in accordance with one or more of the positioning assistanceschemes noted in FIGS. 4-7, a patient may be guided to properly presentthemselves or an object to an image capture device for capture andinterpretation during the noted training phase, or (as will be describedbelow) during a particular medication administration phase. Any of thedisplay and notification techniques noted in any of these Figures may beused in any of the other Figures, in accordance with various embodimentsof the invention. Further, these positioning techniques may be employednot only during initial training, but during any subsequent systemprocess employing video image capture of people, objects, or any otherentity to be imaged, or the use of audio information.

Thus, in one or more situations in which proper positioning of amedication administration device, such as a metered dose inhaler, dryinhaler or the like, is desired to be monitored, the relativepositioning of such a device to the face of a user, for example, may bedetermined in order to determine whether the user is properly holdingand positioning the inhaler for appropriate use. As noted above,improper placement may impede proper medication administration. Throughthe user of computer vision to track and determine proper placement andactuation of the inhaler, best practices can be determined andconfirmed. Any malicious intent on the part of the user may bedetermined based upon purposeful incorrect usage determined based uponvideo or audio signals received and analyzed. Alerts may be sent to theuser warning them on such a problem, or to a clinical trial coordinatoror healthcare provider, as appropriate.

Referring back to FIG. 2, at step 240 these motions of the user may becaptured and confirmed as being correct by one or more appropriatecomputer vision techniques, individual review by a human, or otherappropriate determination process. If not correct, processing may returnto step 230 to provide the instructions and example sequences again tothe user. Therefore, in accordance with the invention, repeatedinstruction may be provided to the patient until training can beconfirmed that the patient has performed the desired sequence correctly,thereby aiding in limiting future variability in the actions taken bythe patient during administration. Such instruction may take the form ofanalysis of a recorded user action, and comments on what the user may bedoing wrong, and how this action may be improved. Once the user hasreceived sufficient instruction, and it is therefore determined that theuser has performed the action in a manner that is sufficiently similarto the instruction set, and substantially consistent over a number ofperformances of the action, processing then passes to step 245 where itis determined whether there are additional training steps to bepresented, and therefore additional video sequences to be captured. Ifso, processing returns to step 220 for further processing. If not,processing ends at step 250.

Referring back to the lower portion of FIG. 1, the horizontal lineindicates a time for patient administration of medication. At such time,the user may be notified to take their medication through any desirablecommunication and notification system, including text messaging, email,telephone call, automated calendar reminder or the like. While notexplicitly shown, first, preferably the identity of a user is confirmedthrough the use of a facial recognition sequence, other biometricidentification sequence, or other password identification system. Uponrecognition of the individual, the system may display one or more dataregarding the individual, such as, by way of example only, name, patientstatus, medication to be administered, calendar indicating to thepatient when medication has been administered and if any administrationtimes have been missed, and, selectively, a score indicative of a levelof compliance of the individual with the medication protocol, ifdesired. Once identified and notified of a type of medication to beadministered, the patient may display a medication administrationapparatus, such as an inhaler, injectable apparatus, or other medicationform (including a pill bottle, pill, or the like) to confirm that themedication is correct and is the currently prescribed medication to betaken through the use of text recognition, medication recognition,barcode or other code reading of one or more unique identifiers from theadministration apparatus, pill bottle or the like, or other appropriatemedication recognition scheme. The user may alternatively be shown avirtual medicine cabinet with visual or textual indications of one ormore medications to be taken, and/or one or more medication apparatusesto be employed, at a particular time. Imaging of one or more of suchmedication apparatuses or medications may then match a medicationapparatus provided by the patient to one or more medications for usewith the matched medication apparatus in the virtual medicine cabinet.Thus, the patient is not only allowed to have a particular medicationapparatus imaged, but also may be given a visual representation ofmedications to be taken, medications that have already been taken, and avisual picture of one or more additional medications to look for if thepatient is confused or is not immediately able to locate all of therequired medication. Such a display may further act as an additionalincentive program for the patient to properly take their medication, andmay in turn give a patient other incentives, such as a running score,payment information or other point systems if the patient is to berewarded for properly taking medication. Thus, credit to buy informationfrom a website or store may be provided. For children, variousanimations may be provided, and pocket money or other credits may beprovided to purchase items online or through one or more stores fromsupporting merchants may be provided. The display of such informationmay assist in convincing the patient to continue to properly takemedication. Such positive reinforcement may include one or moreanimations or the like to encourage desirable (or discourageundesirable) behavior. Posting of success results or other informationto one or more social networking sites may encourage group support toencourage medication. Chat features and the like may be employed forsuch encouragement. Peer networks, or other private groups of patientsmay be set up to allow for more open lines of communication betweenpatients, and to allow for encouragement based upon the activities ofthe patient. Therefore, in accordance with the various embodiments ofthe invention, communication tools, including the ability to accesssocial networking sites, peer to peer networks, private groups and thelike are preferably provided with the monitoring device so that ease ofuse of these types of systems may be provided. This sequence of stepstherefore acts as an audit trail each time a medication is taken, thatcan be reviewed later, to ensure that a patient is properly following aregimen. Any of the positioning schemes depicted in FIGS. 4-7 may beemployed.

Additionally, after confirmation or failure of confirmation of suchadministration, the user may be provided with a progress reportregarding how they have performed over time, and further providingencouragement for future adherence. A point system may be provided inwhich excellent adherence behavior may be rewarded with rewards, such asgifts and the like. These gifts are preferably geared towards particularpatient populations, but may also provide points allowing the user toselect a particular reward. Thus, such rewards encourage good behavior.Additionally, notice that a particular user has received a reward may beprovided to the other users through the social networks or the like,thus encouraging good behavior on their part as well. Such rewards maybe awarded based upon user preferences, or other demographic informationor the like. A lottery type system may be provided in which a number ofusers reaching a particular percentage of adherence, or by some othermetric, may be entered into a lottery for a great reward. Rewards mayalso be somewhat randomized, thus allowing users to be surprised as towhat they may receive. Additionally, notice of a next administrationtime may be provided, along with one or more messages from a healthcareprovider regarding protocol changes, or other desired information.

Furthermore, use of a combination of visual and/or audio cues may beemployed to further determine sequence and timing. Thus, not only shouldan inhaler be properly positioned, for example, but during use, aninhalation by the patient should occur immediately after actuation ofthe inhaler. Thus, by visually and/or audibly confirming firstactuation, and then inhalation, this sequence of actions can beconfirmed. Sound and visual signatures related to each of these actionsmay be employed to improve a confidence with which the system is able toconfirm proper administration. Similarly, an injectable may need to beproperly positioned and maintained in a particular position afteradministration, such as maintenance of a needle after actuation of theinjection mechanism for a predetermined period of time.

In accordance with the invention, confirmation of patient adherence tothe prescribed administration schedule for the medication as prescribedby the clinical trial or other prescription regimen may be determined.While such confirmation may take a number of forms, in accordance withthe invention, a preferred method for such confirmation may includecapturing a video and audio sequence of the patient actuallyadministering the medication. In a further preferred method, such asequence for such confirmation may include employing a facialrecognition sequence or other biometric confirmation that a particularpatient is in fact receiving treatment, but may also provide for theability to obscure the face or other identifying feature of a user, orotherwise encrypt such information to allow for the storage and use ofsuch images while protecting the identity of the patient, a techniquethat may be beneficial when a medication administration manager isproviding a general report about a clinical trial, and not trying toremedy a situation with a particular patient, or in particular in apublic health or disease management scenario. Activity recognition,gesture recognition, utilizing computer vision or the like, or otherfeature for determining whether a particular subject movement meets apredefined movement sequence may be employed to be sure that the patientis properly taking prescribed medication.

Referring next to FIG. 3, a method in accordance with an additionalembodiment of the present invention for performing audio and videocapture and recognition of adherence to a prescribed protocol isdescribed, as set forth in steps 130 and 135 of FIG. 1. In FIG. 3, apatient may first log into the system of the invention at step 305,employing the facial recognition, biometric recognition, password entry,or other patient identification method, and at step 310 propermedication is confirmed as noted above, through the user of bar codereading, text recognition, visual recognition employing video or stillimage recognition, or other medication recognition technique. Thepatient may be reminded to log onto the system to take their medicationthrough any type of reminder, such as a text message, email, phone call,automated alarm or the like. Of course, any of the positioningtechniques previously described in reference to FIGS. 4-7 may beemployed. Next, at step 315 it may be confirmed that the processinvolved will include one or more information capture steps, and at step320 it may be determined whether these information steps will includevideo capture. If not, video processing ends after storage of anynon-video information. (Alternatively, steps 315 and 320 may be excludedif it is determined that each confirmation sequence may employ videocapture, then video processing may pass directly to step 325, asdescribed below.) If it is confirmed at step 320 that one or more stepswill include video and/or audio capture, processing then passes to step325 where the user may be prompted to perform one or more predeterminedactions, these actions being captured. Positioning of the inhaler,injectable medication apparatus, or other medication may be performed inaccordance with any of the techniques as described previously inreference to FIGS. 4-7. Such recognition in the case of an injectable orinhalable administration apparatus may also comprise confirmingrelationship of the injectable or inhalable administration apparatus anda prescribed body part, proper actuation of the administrationapparatus, including inhalation in the case of an inhalableadministration apparatus, maintaining the administration apparatus inthe location for a predetermined period of time, and perhaps proper postadministration action, such as cleaning and storing the apparatus,refrigerating the apparatus, cleaning an injection site and the like.Further, voice recognition may be utilized to allow the user to entercommands, and an audio output may be provided for aiding the user inproperly adhering to instructions from the system. Additional audio cuesmay be recognized, such as upon visual confirmation of administration ofan injectable or inhalable medication, audio signatures may be employedin order to determine whether insufficient pressure may have been used,or whether a sufficient or extensive period of time has passed fromactuation to inhalation. Additionally, audio signatures may be employedto confirm that the inhalable material was properly inhaled, and notblocked, for example, by the teeth of the user. Proper capture ofpatient actions is very important as the patient only administers themedication once per capture period.

As is further described below, such audio inhalation may be employedwith micro lung movement recognition to further improve confidence thata correct administration action has taken place. It is thereforecontemplated in accordance with an embodiment of the invention that anysuch video or visual analysis may include the use of micro movements toaid in determining inhaler or other apparatus actuation. In particular,small movements in the jaw or chest may be employed to aid indetermining whether or not medication from an actuated inhaler isactually being inhaled. Any number of such micro movements may beemployed, from movement of one or more portions of the medicationadministration device, and body parts that are to move based upon propermedication administration or the like. Such body part movement mayadditionally include, in the case of use of an inhaler, movement of thehead of the user, movement of the hand of the user. Changes in therelative positioning of the inhaler and the mouth or other facialportion of the user, and the like. These micro movements may be usedalone, or in combination with other visual and audio cues to confirmproper medication administration. Further, such movements may beemployed to aid in determining an amount of medication that has beeninhaled, or otherwise ingested. Audio and visual duration of inhalationand micro movements may be considered objectively, and in relation toprior inhalation activities of a user to determine changes in patternsthereof. A measured inhalation system may be first provided to allow thesystem to “learn” how a proper inhalation looks and sounds, using thisas a reference to judge later inhalation instances. Various demographicand other patient related data may further be employed in order todetermine an amount of medication that should be inhaled or otherwiseingested or injected by a user. Thus, a larger individual should have alonger inhalation cycle, and a deeper inhalation breath, for example,than a smaller child. Other demographic information , such as age, etc.may also be used to properly calibrate the systems.

Video capture analysis may then begin at step 330, such analysiscomprising analysis of the newly captured video and/or audio, asprovided as noted above with respect to FIG. 2. At step 335 it may bedetermined whether the action has been properly captured, and whetherthe captured action has been properly analyzed by the system. Variousincentives may be provided to the patient to encourage them to taketheir medication properly. Thus, in addition to providing variousreminders to a patient as is known in the art, points, monetary or otherincentive may be provided to the user for actually having medicationadministration confirmed. Further proper administration with lesserrors, etc. may be rewarded more highly, thus giving incentive for thepatient to concentrate on administration issues and to attempt to havesuch administration be as accurate and consistent as possible. Suchincentives and medication tracking may be used to determine futurecourses of treatment or payment. For example, if a patient consistentlyfails to take medication as required, perhaps a different course oftreatment requiring fewer medication administrations may be better forthis patient. Alternatively, if a medication requires a consistentadministration and is very expensive, failure to comply withadministration instructions may be cause for an insurance company,prescribing doctor or the like to not renew such a prescription for thepatient, thus saving money in a situation where the money was beingwasted because of lack of compliance.

If it is determined that administration of the medication did not takeplace properly, processing may return to step 325 and the user may beonce again prompted to perform the action. Of course, if this processinvolves actual administration of inhaler or injectable medication, itmay not be proper to request re-performance of the action, unless it canbe determined that the user did not actually administer the medication.If the action has been properly captured, and is able to be analyzed,processing passes to step 345 where it may be determined whetheradditional captures are required. If so, processing returns to step 320.If no further captures are required, processing ends at step 350 wherethe various captured video sequences are stored. These stored sequencesmay also be made available for human review and involvement, when it isdetermined that this would be beneficial.

Therefore, in accordance with various embodiments of the invention,because a video image of the patient actually administering an inhalableor other medication (or other method of medication administration,including but not limited to injections, dialysis, and any othermedication administration procedure) may be captured and analyzed,actual confirmation may be achieved, rather than simply relying on thepatient to state that a particular medication was administered. Such avideo image may be captured or stored in any appropriate format given aselected type of activity or gesture recognition that is employed inaccordance with a particular embodiment of the invention. Such mayinclude full video, biometric data points, recording of movement of anarticle, such as a bracelet or the like, affixed to the patient oradministrator, use of mapping to provide a stick figure or other bodymovement tracking technique, or gesture or activity recognition todetermine movement or the like. The user may be encouraged to use aparticular sequence of movement to be confirmed that they are properlyadministering the medication according to the protocol, thus reducingthe possibility of the potential appropriate movements considered to be“correct.” Or, as noted above, capture of customized video sequences maybe performed so that the user is more likely to repeat these sameactions. Indeed, various instructional videos or other appropriatetraining may be provided to a user to insure they properly administerthe medication.

This captured adherence information may be provided to a healthcareprovider, clinical trial manager or the like through a dashboardallowing for the review of information about an individual patient,entire population of patients, or demographically relevant information.Such information may be provided to easily notify the healthcareprovider, clinical trial manager or the like of problem patients,demographic groups, medications or the like. One or more dashboards orother reporting mechanisms may be employed as described in copendingU.S. patent application Ser. No. 13/189,518, filed Jul. 24, 2011 toHanina et al., titled “Method and Apparatus for Monitoring MedicationAdherence”, the entire contents thereof being incorporated herein byreference. Thus, any adherence or other information obtained inaccordance with the present invention may be provided to one or moreindividuals in accordance with one or more methods or systems asdescribed in the '518 application.

Through the use of training as described above, a type of administrationlanguage may be generated, allowing for extension to other patients, andalso allowing for interpretation of reason for differences from apredefined sequence by a patient. Thus, if a patient performs an actiondifferently over time, this difference may provide insight to a reactionto a medication, changes in the patient's medical condition, or thelike. It is further anticipated that analysis of large numbers ofpatients will allow for a more flexible system that may recognize moreof a patient's movements, and thus may improve the ability of the systemto function properly.

Therefore, in accordance with an embodiment of the invention, a user mayperform a predetermined sequence of actions designed to ensureperformance of medication administration. Thus, by way of example only,for an inhaled medication as noted above, the user may be asked to firstshow a medication and may then be prompted to position the medicationadministration apparatus relative to their mouth in a desired manner.Next the user may be prompted to administer the medication, the actionof administration being captured on video and audio, and beinginterpreted to confirm that the medication has been properlyadministered. Of course, in accordance with this embodiment of theinvention, other action sequences may be employed, and may be mixed withother actions to be performed by a patient or caregiver. Thus, butdefining a medication adherence protocol as a single or sequence ofgestures that may be recognized by a processing system, the accuracy ofconfirming that a patient has actually taken a particular medication isimproved. Through an interactive learning process, the processing systemmay also learn patient behaviors to be more accurately determinemedication adherence, and to remove some of the potential falsepositives or false negatives. If a caregiver is involved, it iscontemplated that the caregiver be provided with a number of gesturesindicative of particular actions to be taken, and use of these gesturesprompting the system to confirm that these actions are in fact beingtaken. Thus, a full audit trail of not only the patient, but also thecaregiver may be determine, such as whether they approached the patientat the correct times, or that they washed their hands when approaching.

Further in accordance with an inhalable medication administrationapparatus, such as a metered dose inhaler or dry inhaler, the use ofaudio signatures in addition to visual cues may be beneficial.Therefore, in accordance with an embodiment of the invention, aninventive audio cue recognition system may be employed. In such anembodiment, a standard audio analysis and recognition system may beemployed. Such an analysis system can typically easily distinguishsounds associated with proper administration of inhalable medication ina relatively quiet environment (i.e. an environment without substantialbackground noise). The inventors of the present invention havedetermined that such a system has a high accuracy rate of about 95%, andonly includes a 1% false positive error. Such an analysis systemincludes to modules, a first for feature extraction of audio features,and a second for classification of the extracted sounds. Preferably thefeature extraction module may employ Mel-Frequency Cepstrum Coefficients(MFCC) as feature points. Thereafter, SVM machine learning may beemployed for the classification module. While the system is relativelysimple and fast, and thus easy to integrate into other systems and iseasily used in real time, it may not be sufficiently robust in a noisyenvironment.

Therefore, in addition to this type of audio recognition system, theinventors of the present invention present a multiple level soundrecognition system, for use of detection of audio signatures inaccordance with inhaler use in the present invention. Of course, such amultiple level sound recognition system may be employed in any medicalor other application. Therefore, in accordance with this embodiment ofthe invention, a first level recognition is performed similar to that ofthe above embodiment in which sounds are determined to have passed afirst filter. It is desirable to set this level to be inclusive, ratherthan exclusive, so that sounds that are clearly no related to inhaleraudio signatures be excluded, but sounds that may be audio signaturesare included. Such recognition system may also be employed to othersound in administration, such as confirming proper installation of amedication canister, for example.

After determining a plurality of signals that have passed the firstlevel filter, these passed sounds are then preferably used to train asecond level filter. Because all of these sounds have passed the firstlevel filter, the sounds will be relatively similar. Thus, because therange of sounds will be narrower, the precision of this second levelfilter will be far greater. By training the system to recognize inhalersounds form background noise, only employing sounds that passed a firstlevel filter, precise determination of inhaler sounds can be achieved.Of course, and number of levels may be employed in order to improveprecision of the system, but at the expense of training and processingtime.

FIG. 8 depicts operation of a combined video and audio detection systemfor detecting proper actuation and use of an inhaler medication. As isshown in FIG. 8, a step 805 asks the user to position their face isprovided, allowing for facial identification or the like as describedabove. During this time, an audio detection system is preferablyreceiving sound in a microphone at step 810, is performing anenvironmental check to determine whether there is too much noise toperform a proper audio analysis at step 820, and if it is too noisy,issuing a warning at step 824. Once the face positioning and facialrecognition step is completed, processing on the video side passes tostep 815, where proper positioning of the inhaler apparatus isperformed, utilizing one or more of the positioning systems describedabove. In addition to proper positioning, this step may also be used toproperly identify the inhaler, medication or the like. During this timeframe, steps 810, 820 and 824 as described above are performed tocontinue to look out for situations in which there are too much noise toperform a proper audio analysis.

The dashed line in FIG. 8 depicts a time for providing an instructionfor actuation of the inhaler device. Thus, on the video side, at step825 proper actuation of the inhaler based upon video evidence isprovided. Procedures for this have been described above, and will bedescribed in greater detail below. At the same time, step 830 isperformed on the audio side to make a determination of whether an audiosignature is indicative of proper actuation of the inhaler apparatus.Details of this audio determination will be described below. Upon properdetermination of actuation of the inhaler apparatus by video in step 825and by audio in step 830, it can be determined, to some level ofconfidence, that the inhaler apparatus has been properly actuated.

Referring next to FIG. 9, the audio signal confirmation steps shown inFIG. 8 in accordance with an embodiment of the present invention willnow be more particularly described. In the process for determiningmedication adherence, and during video steps before inhaler apparatusactuation, sound is received at a microphone at step 910. This soundinformation is stored in a byte array at step 912. Next, at step 920, anenvironmental checker performs a check of the noise in the signal. If atstep 922 it is determined that that it is too noisy, a warning isprovided at step 924 to instruct the user that it is too noisy, and tofind another location that is perhaps less noisy. Processing thenreturns to step 910 to receive additional sound to determine whether thesound situation has improved. If at step 922 it is determined that it isnot too noisy, then processing similarly returns to step 910 to continueto listen for sound to determine whether the sound environment hasdeteriorated.

After steps 805 and 815 in FIG. 8 have been completed, the system willinstruct the user to actuate the inhaler apparatus, a time indicated bythe dashed line in FIG. 9. At this time, processing will pass from step922 to step 930 where a sound of actuation is recorded, and the dataassociated with such sound of actuation is normalized and typetransformed. Processing then passes to step 940 where the sound ispreprocessed. Then, at step 950, a first level SVM model is applied tothe data (as described above) to determine whether a recorded sound ispossibly a target sound. If the inquiry at step 955 is answered in thenegative, the recorded sound is determined not to be the target sound,and it is concluded that the sound is not that of an inhaler at step957.

If on the other hand, the inquiry at step 955 is answered in theaffirmative, and it is determined that the recorded sound is possiblethe target sound, processing then passes to step 960 in which a secondlevel SVM model is applied (as described above). This second levelmodel, as noted, is more precise than the first level, and is able to doso because all clearly incorrect sounds have been removed. Thus, thissecond level processing can be more precise and sensitive, focusing onnuanced differences between the various sounds. At step 965, it isinquired whether this second level SVM model has indicated that therecorded sound is the target sound. If this inquiry is answered in thenegative, and it is determined that the sound is therefore not the soundof the actuation of an inhaler, such an indication is made at step 967.If on the other hand, the inquiry at step 965 is answered in theaffirmative, and it is once again determined that the recorded wound issimilar, to the desired level of precision, to the target sound, it isindicated at step 970 that an inhaler actuation sound has been detected.This result, along with any results from the video analysis system, maytherefore be used to determine proper inhaler actuation.

Additional features may be employed in accordance with encouragement anddetermination of proper inhaler actuation. In one particular embodiment,if a particular amount of time is to pass after inhalation beforeexhalation, a countdown timer or other time measuring processing may bedisplayed on the display of a monitoring device to properly instruct theuser to hold their breath. Such graphical or audible information may betailored to the population at issue, such as a whimsical treatment maybe provided for a younger, child population, while a more standardcountdown clock may be provided for adults.

In addition to simply determining whether the recorded sound (recordedin step 930 of FIG. 9) is an inhaler actuation, it may be desirable tofurther segment this determination into classifications of inhaleractuation. The inventors of the present invention have determined thatone error often made by users of inhaler devices is to sometimes blockthe flow of air and medication with their teeth. Therefore, inaccordance with an alternative embodiment of the invention, anotherlevel may be provided to determine a difference between actuation withteeth in the way and actuation without teeth in the way. By training theaudible recognition system to differentiate between these two sounds,proper identification can be determined. In such a manner, if it isdetermined that the teeth of the user are in the way, the user may becongratulated for using the device, and provided further instruction toremove their teeth from the flow of air. These two determinations willplace the system in different states, and any desired action may beperformed based upon the current state of the system.

Similar systems may also be employed to determine, via audio and videocues, whether the inhaler has been held incorrectly, or is being heldaway from the user's mouth, for example. Such audio may aid when acamera placed directly in front of a user, may still recognize a properinhalation. Changes in perspective for the video system may also besuggested based upon past and/or current audio and video analysis. Thus,alerts may be more confidently employed in accordance with misuse of thesystem when the inhaler or other device is improperly used.

In an alternative embodiment of the system, noise cancellation may beemployed in accordance with the environmental checker of FIGS. 8 and 9.Thus, rather than simply rejecting a particular environment as toonoisy, an analysis of the noise characteristics may be provided. Thisanalysis may be further employed to determine whether a noisecancellation operation may reduce background noise to a point where suchenvironment may be acceptable. If so, a warning may still be provided,or alternatively, the warning may be overridden as the system mayoperate properly with use of the noise cancellation system.

In addition to segmenting the inhalation audio signature into twoclasses, one with teeth in the way and one without teeth in the delay,any number of additional classifications may be provided. In aparticular embodiment of the invention, the amount of air inhaled may besegmented, so that, for example, the sound of a high volume of airintake may be differentiated from the sound of a low volume of airintake. This may be valuable to determine objectively whether enough airhas been inhaled, or may be used in a sequential manner to determinewhether an air intake profile for a particular user changes over time,or from one administration to another.

Further uses of the video capture sequences may also be employed,including video capture of responses to questionnaires about currentpatient states of discomfort, informed consent, example of questions tobe asked, video transmission of such questions and the like. The patientmay be able to send a video message, pointing to a particular pain orthe like, and may include an audio portion as well. Time stamp markersmay also be captured to confirm that the user is taking their medicationat appropriate times and a number of times a user has taken a particularmedication, to confirm whether there are substantial delays betweeninstruction and administration, or for any other time sequencedetermination. Furthermore, other behavioral markers, such as, by way ofexample only, shaking hands indicating a particular ailment, or othermovements by a patient that may give a hint as to the physical or mentalstatus thereof. Additionally, if the user is taking medication that isimproper, or they have already taken, a warning may be provided to warnthe user to stop medication administration immediately.

In accordance with various embodiments of the invention, whenconsidering administration of an inhalable or injectable medication,analysis of adherence video sequences may be employed to determine alikelihood that a patient has actually administered their medication.Thus, based upon video and audio cues determined related to positioningand use of the medication administration apparatus, it may be determinedthat the patient is having problems properly positioning the apparatus,and therefore the system is unsure that the patient has administered themedication properly. Low confidence in proper administration based uponfailure to properly position the apparatus, failure to record audiosignals indicative of proper administration or the like may be employedto determine whether a patient should be retrained, via the automatedtraining system described herein, by automated contact, or by individualpersonal contact. This determination of low confidence ofadministration, even if it is ultimately determined that administrationlikely took place, may still be utilized to determine whether trainingor other actions may be taken. Such confidence levels may be used, inaccordance with a desired algorithm or the like, to provide an overallpicture of medication administration by a patients or group of patients,thus allowing for intervention, encouragement, training or the like tobe provided when it appears that actions are changing, but notnecessarily waiting until a critical issue is discovered. Furthermore,if low confidence is determined, further, more robust continuoustracking systems may be employed to confirm that the hand of the usermoves correctly, that the inhaler is properly placed in the hand of theuser, that the inhaler is placed properly relative to the face and mouthof the user, that the inhaler is properly placed relative to the lips ofthe user, and that the canister is properly placed within the inhaler.

It is further contemplated that the method and apparatus of theinvention allow for integration with one or more audio or videoconferencing systems, thus receiving and/or providing information therethrough. Thus, a user may employ a standard video conferencing tool orsystem, and have this information be coupled to a mobile or other devicebeing used in accordance with an embodiment of the present invention.

Therefore, in accordance with the invention, a method and apparatus areprovided that allow for the automated confirmation of adherence toadministration protocol for medication, and provide for a mostsophisticated method for confirming and studying methods ofadministration of such prescription medication.

It will thus be seen that the objects set forth above, among those madeapparent from the preceding description, are efficiently attained and,because certain changes may be made in carrying out the above method andin the construction(s) set forth without departing from the spirit andscope of the invention, it is intended that all matter contained in theabove description and shown in the accompanying drawings shall beinterpreted as illustrative and not in a limiting sense.

It is also to be understood that this description is intended to coverall of the generic and specific features of the invention hereindescribed and all statements of the scope of the invention which, as amatter of language, might be said to fall there between.

1. A medication administration confirmation apparatus for confirmingadministration of medication employing an inhalable medicationadministration apparatus, comprising: a video capture device forcapturing one or more video sequences of a user administering medicationemploying the inhalable medication administration apparatus; a memoryfor storing the captured one or more video sequences; an audio capturedevice for capturing one or more audio sequences of the useradministering medication employing the inhalable medicationadministration apparatus; a memory for storing the captured one or moreaudio sequences; and a processor for analyzing at least one of thestored video sequences and at least one of the stored audio sequences toconfirm that the user has properly administered the medication.
 2. Themedication confirmation apparatus of claim 1, wherein the user isprompted to perform a particular sequence of actions to be stored as theone or more video sequences.
 3. The medication confirmation apparatus ofclaim 2, wherein the one or more audio sequences are stored inaccordance with the performance of the particular sequence of actions.4. The medication confirmation apparatus of claim 2, wherein theparticular sequence of actions is designed to assist in determiningproper medication administration.
 5. The medication confirmationapparatus of claim 1, wherein the one or more audio sequences areanalyzed to determine their similarity to a target audio sequence. 6.The medication confirmation apparatus of claim 5, wherein the analysisis performed in a two level process, comprising: a first step ofdetermining whether one of the one or more captured audio sequencespasses a first filter as being similar to a target audio sequence; andafter passing the first step, a second step for determining whether theone or more captured audio sequences passes a more stringent filter asbeing similar to the target audio sequence.
 7. The medicationconfirmation apparatus of claim 6, wherein the more stringent filter ofthe second step is trained based upon audio sequences that pass thefilter in the first audio step.
 8. The medication confirmation apparatusof claim 1, wherein the analysis confirms proper positioning of theinhalable medication administration apparatus.
 9. The medicationconfirmation apparatus of claim 1, wherein the apparatus is employedduring a training sequence.
 10. The medication confirmation apparatus ofclaim 1, wherein the processor analyzes the at least one stored videosequences to determine one or more micro movements of the user.
 11. Themedication confirmation apparatus of claim 1, wherein the processoranalyzes the one or more captured audio sequences to determine an amountof medication inhaled by the user.
 12. The medication confirmationapparatus of claim 1, wherein the processor analyzes the one or morecaptured audio sequences to determine whether the teeth of the user arein the path of inhalation.
 13. The medication confirmation apparatus ofclaim 1, further comprising an environmental checker to determinewhether there is too much background noise to properly capture the oneor more audio signals.
 14. The medication confirmation apparatus ofclaim 13, wherein the processor further applies a noise cancellationalgorithm to the one or more captured audio signals.
 15. A medicationconfirmation method for confirming administration of medicationemploying an inhalable medication administration apparatus, comprisingthe steps of: capturing one or more video sequences of a useradministering medication employing the inhalable medicationadministration apparatus; storing the captured one or more videosequences; capturing one or more audio sequences of the useradministering medication employing the inhalable medicationadministration apparatus; storing the captured one or more audiosequences; and analyzing at least one of the stored video sequences andat least one of the stored audio sequences to confirm that the user hasproperly administered the medication.
 16. The method of claim 15,further comprising the step of prompting the user to perform aparticular sequence of actions to be stored as the one or more videosequences.
 17. The method of claim 16, further comprising the step ofstoring the one or more audio sequences in accordance with theperformance of the particular sequence of actions.
 18. The method ofclaim 15, further comprising the step of analyzing the one or more audiosequences to determine their similarity to a target audio sequence. 19.The method of claim 18, wherein the analysis is performed in a two levelprocess, further comprising: a first determining step for determiningwhether one of the one or more captured audio sequences passes a firstfilter as being similar to a target audio sequence; and after passingthe first determining step, a second determining step for determiningwhether the one or more captured audio sequences passes a more stringentfilter as being similar to the target audio sequence.
 20. The method ofclaim 19, wherein the more stringent filter of the second step istrained based upon audio sequences that pass the filter in the firstaudio step.